Summary

This PR fixes the issue where npm publish failures were incorrectly reported as successes. The root cause was that changeset publish exits with code 0 even when packages fail to publish, and the script only relied on exception handling for failure detection.

Changes:

• Multi-layer failure detection in publish-to-npm.mjs:

  • Output pattern matching for known error messages (packages failed to publish, npm error code E, E404, E401, E403, Access token expired, ENEEDAUTH)
  • Exit code checking (when available)
  • Post-publish verification by querying npm registry
  • Retry logic now properly triggers on all failure types

• Case study documentation in docs/case-studies/issue-115/:

  • Detailed timeline of the original failure
  • Root cause analysis with error chain diagram
  • References to upstream changesets issues (#1621, #1280)

• Test script in experiments/test-failure-detection.mjs:

  • Validates failure detection patterns against real CI output

Root Cause

The original issue occurred because:

1. changeset publish exits with code 0 even when packages fail to publish (known upstream issue)
2. The command-stream library doesn't throw exceptions on non-zero exit codes
3. The script only used try-catch for failure detection

How it's Fixed

The script now:

1. Captures the output of changeset publish using .run({ capture: true })
2. Scans output for failure patterns
3. Checks exit code (for defense in depth)
4. Verifies the package is actually on npm after publish (ultimate verification)
5. Only reports success after verification passes

Test Plan

• Lint passes (npm run lint)
• Format check passes (npm run format:check)
• Unit tests pass (bun test tests/json-standard-unit.test.js)
• Failure detection test passes (node experiments/test-failure-detection.mjs)

Related Issues

Fixes #115

🤖 Generated with Claude Code